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DETAILED ACTION 

This action is in response to the amendment filed on 21 July 2010. 

EXAMINER'S AMENDMENT 

1 . An examiner's amendment to the record appears below. Should the changes 
and/or additions be unacceptable to applicant, an amendment may be filed as provided 
by 37 CFR 1 .312. To ensure consideration of such an amendment, it MUST be 
submitted no later than the payment of the issue fee. 

Authorization for this examiner's amendment was given in a telephone interview 
with Albert W. Watkins on 30 August 2010, September 24, 2010 and September 27, 
2010. 

The application has been amended as follows: 

a. The title will be changed as follows: 

A Proqrammorator Method For Monitoring A Graphical User Interface On A 
Second Computer Display From a First Computer 

b. The claims will be amended as follows: 

Claim 8. A method for using a first computer system to remotely monitor 
and interact with the operation of a second computer system through a 
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graphical user interface of said second computer system, comprising the 
steps of: 

receiving a pixel bitmap image of said second computer system 
graphical user interface at said first computer system; 

searching said pixel bitmap image of said second computer system 
graphical user interface for a first graphical element wh i ch may 
b e found at on e an i nd e term i nat e l ocat i on contained within and 
comprising less than said pixel bitmap image through an 
automated execution of said first computer system commands; 

responsive to said receiving step and results of said searching step , 
generating a user peripheral input device input action within said 
second computer system graphical user interface as interpreted 
by a said second computer p e r i ph e ra l i nput d e v i c e contro lle r 
channe l by automatically creating and passing a signal through 
aft i/e ^communications channel from said first computer system 
to said second computer system graphical user interface 
r e spons i v e to sa i d r e c e iv i ng st e p and r e su l ts of sa i d s e arch i ng 

monitoring said pixel bitmap image of said second computer system 
graphical user interface automatically from said first computer 
system for an expected second graphical element contained 
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within and comprising less than said pixel bitmap image within a 
predetermined time interval; and 
signaling a failure at said first computer system if said predetermined 
time interval elapses without detecting said expected second 
graphical element. 

Claim 9 will be canceled. 

Claim 1 1 . The method of claim 8 further comprising the steps of: 

generating a user input action within said second computer system 

responsive to said second graphical element; 
monitoring and searching said second computer system graphical 
user interface automatically from said first computer system for 
an expected third graphical element wh i ch may bo found at ono 
aft i nd e t e rm i nat e l ocat i on contained within and comprising less 
than said pixel bitmap image within a predetermined time 
interval; and 

signaling a failure at said first computer system if said predetermined 
time interval elapses without detecting said expected third 
graphical element. 
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Claim 14. A method for enabling a local system to automatically remotely 
operate a remote computer system through a graphical user interface on 
said remote computer system by using local scripts that selectively 
respond to changes in graphical displays upon said graphical user 
interface of said remote computer system, comprising the steps of: 

displaying a depiction of said remote system graphical user interface 

display on said local system; 
capturing user input effected in said depiction of said remote system 

graphical user interface display; 
implementing automatically through a local system command 
language set user input emulations that are representative of 
said captured user input when reproduced at said remote 
computer system graphical user interface through a peripheral 
input device ife channel; 
image processing said remote computer system graphical displays 
automatically using a local system script that searches for and 
detects a first entity wh i ch may b e found at an i nd e t e rm i nat e 
l ocat i on contained within and comprising less than said graphical 
display upon said graphical user interface of said remote 
computer system; 
controlling a flow of execution of said local system automatically 
through a scripting language having scripting commands in 
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combination with said command language set, said flow which 
varies responsive to a result of detection of said first entity during 
said image processing step; and 
communicating said user input emulations between said local 
system and said remote computer system graphical user 
interface through a communication interface responsive to said 
flow controlling step. 



Claim 16. A method for using a first computer system to remotely monitor 
and interact with the operation of a second computer system through a 
graphical user interface of said second computer system, comprising the 
steps of: 

receiving a representation of said second computer system graphical 
user interface at said first computer system; searching said 
representation of said second computer system graphical user 
interface for a first graphical entity wh i ch may bo found at ono of 
aR i nd e t e rm i nat e l ocation contained within and comprising less 
than said representation of said second computer system 
graphical user interface through an automated execution of said 
first computer system commands; and 

responsive to said receiving step and results of said searching step, 
generating a user peripheral input device input action within said 
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second computer system graphical user interface as interpreted 
by a second computer per i phera l i nput dov i co contro ll er channe l 
by automatically creating and passing a signal through an i /o a 
communications channel from said first computer system to said 
second computer system graphical user interface r e spons i v e to 
sa i d roco i v i ng stop and resu l ts of sa i d scorch i ng stop ; 

Claim 19. The method of claim 37 further comprising the steps of: 

generating a user input action within said second computer system 

responsive to said second graphical entity; 
monitoring and searching said second computer system graphical 
user interface for an expected third graphical entity within a 
predetermined time interval; and signaling a failure at said first 
computer system if said predetermined time interval elapses 
without detecting said expected third graphical entity. 

Claim 28. A method for enabling a first computing machine to remotely 
operate a second computing machine through a graphical user interface 
on said second computing machine by using computing scripts that 
selectively respond to changes within graphical displays upon said 
graphical user interface of said second computing machine, comprising 
the steps of: 
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displaying a depiction of said second computing machine graphical 
user interface display on a graphical user interface of said first 
computing machine; 

capturing user input effected in said depiction of said second 
computing machine graphical user interface display; 

image processing said second computing machine graphical 

displays using a first computing machine script that searches for 
and detects the presence of a first entity wh i ch may bo found at 
l ocat i ons an i nd e t e rminat e l ocat i on contained within and 
comprising less than said graphical display upon said graphical 
user interface of said second computing machine; 

controlling a flow of execution of said first computing machine 
through a scripting language having scripting commands in 
combination with said a command language set, said flow which 
varies responsive to a result of detection of said first entity during 
said image processing step; and 

implementing user input commands at said second computing 
machine graphical user interface emulating said captured user 
input by communicating between said first computing machine 
and said second computing machine graphical user interface 
through a communication interface responsive to said flow 
controlling step, a first user input command implemented 
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responsive to a detection of said first entity during said image 
processing step and a second user input command different from 
said first user input command implemented responsive to a non- 
detection of said first entity during said image processing step. 

Claim 37. The method of claim 16, further comprising the steps of: 

monitoring and searching said second computer system graphical 
user interface automatically from said first computer system for 
an expected second graphical entity within a predetermined time 
interval; and 

signaling a failure at said first computer system if said predetermined 
time interval elapses without detecting said expected second 
graphical entity. 

Claim 38. A method for using a first computer system to remotely monitor 
and interact with the operation of a second computer system through a 
graphical user interface of said second computer system, comprising the 
steps of: 

receiving a pixel bitmap image of said second computer system 
graphical user interface at said first computer system; 

searching said pixel bitmap image of said second computer system 
graphical user interface for a first graphical element wh i ch may 
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b e found at on e of a p l ura li ty of pot e nt i a l and i nd e t e rm i nat e 
l ocat i on s contained within and comprising less than said pkel 
bitmap image through an automated execution of said first 
computer system commands; 
responsive to said receiving step and a not-found result of said 
searching step, generating a first user peripheral input device 
input action within said second computer system graphical user 
interface by automatically creating and passing a signal through 
an-i/e ^communications channel from said first computer system 
to said second computer system graphical user interface 
r e spons i v e to sa i d r e c ei v i ng st e p and a not found r e su l t of sa i d 
search i ng stop ; 

and responsive to said receiving step and a found result of said 
searching step, generating a second user peripheral input device 
input action within said second computer system graphical user 
interface different from said first user peripheral input device 
input action by automatically creating and passing a signal 
through an-i/o a_communications channel from said first computer 
system to said second computer system graphical user interface 
respons i ve to sa i d rocoiving stop and a found resu l t of sa i d 
search i ng stop . 
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Claim 39. The method for using a first computer system to remotely 
monitor and interact with the operation of a second computer system 
through a graphical user interface of said second computer system of 
claim 38, further comprising the steps of: 

monitoring and searching said pixel bitmap image of said second 
computer system graphical user interface automatically from said 
first computer system for an expected second graphical element 
contained within and comprising less than said pixel image within 
a predetermined time interval; and 
signaling a failure at said first computer system if said predetermined 
time interval elapses without detecting said expected second 
graphical element. 

Claim 40. A method for enabling a first computing machine to remotely 
operate a second computing machine through a graphical user interface 
on said second computing machine by using computing scripts that 
selectively respond to changes within graphical displays upon said 
graphical user interface of said second computing machine, comprising 
the steps of: 

displaying a depiction of said second computing machine graphical 
user interface display on a graphical user interface of said first 
computing machine; 
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image processing said second computing machine graphical 
displays using a first computing machine search command set 
that searches for and detects the presence of a first entity wh i ch 
may bo found at ono of a p l ura li ty of potent i a l and i ndotorm i noto 
l ocat i on s contained within and comprising less than said 
graphical display upon said graphical user interface of said 
second computing machine; 

controlling a flow of execution of said first computing machine 
through a scripting language having scripting commands in 
combination with a command language set, said flow which 
varies responsive to a result of detection of said first entity during 
said image processing step; and 

communicating between said first computing machine and said 
second computing machine graphical user interface through a 
communication interface responsive to said flow controlling step, 
a first user input command implemented at said second 
computing machine responsive to a detection of said first entity 
during said image processing step and a second user input 
command different from said first user input command 
implemented at said second computing machine responsive to a 
non-detection of said first entity during said image processing 
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step. 

Claim 41 . The method for enabling a first computing machine to remotely 
operate a second computing machine through a graphical user interface 
on said second computing machine of claim 40, wherein said first and 
second user input commands are implemented at said second computing 
machine graphical user interface by transmitting signals from said first 
computing machine to said second computing machine graphical user 
interface through a peripheral input device \f& channel. 

42. A method for enabling a first computing machine to remotely operate a 
second computing machine through a graphical user interface on said 
second computing machine by using computing scripts that selectively 
respond to changes within graphical displays upon said graphical user 
interface of said second computing machine, comprising the steps of: 
displaying a depiction of said second computing machine graphical 
user interface display on a graphical user interface of said first 
computing machine; 
capturing user input effected in said depiction of said second 

computing machine graphical user interface display; 
developing at least one computing script incorporating user input into 
said first computing machine selected from ones of various 
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scripting functions and commands, said at least one computing 
script further incorporating said captured user input; 

image processing said second computing machine graphical 
displays using a first computing machine search command set 
that searches for and detects the presence of a first entity wh i ch 
may bo found at ono of o p l ura li ty of potent i a l and i ndotorm i not.0 
l ocat i on s contained within and comprising less than said 
graphical display upon said graphical user interface of said 
second computing machine; 

controlling a flow of execution of said first computing machine 

through said at least one computing script, said flow which varies 
responsive to a result of detection of said first entity during said 
image processing step; and 

implementing user input commands at said second computing 
machine graphical user interface emulating said captured user 
input by communicating between said first computing machine 
and said second computing machine graphical user interface 
through a communication interface responsive to said flow 
controlling step, a first user input command implemented at said 
second computing machine graphical user interface responsive 
to a detection of said first entity during said image processing 
step and a second user input command different from said first 
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user input command implemented at said second computing 
machine graphical user interface responsive to a non-detection of 
said first entity during said image processing step. 



Claim 48. The method of claim 8, further comprising the step of converting 
said received pixel bitmap image of said second computer system 
graphical user interface into a variant different from said second computer 
system graphical user interface prior to said searching step. 



Disclosure of the Prior Art of Record 



Tuttle et al. US 5,157,782: Tuttle discloses creating a script of I/O commands on a first 
computer. Running the script such that the I/O commands are transmitted across an 
I/O channel to a second computer, wherein the I/O commands are input to a software 
application. The application on the second computer outputting a Graphical User 
Interface (GUI) on a display of the second computer. Capturing the digital images 
displayed on the display of the second computer and transmitting the digital images to 
the first computer and storing the digital images at the first computer. Re-executing the 
script on the first computer in the manner described above, and recapturing the images 
on the display of the second computer and transmitting the images to the first computer. 
Comparing the second set of screen images with the first set of images. Signaling an 
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error for each first digital image that does match a corresponding digital image of the 
second set of images. 

Perholtz et al. US 5,732,212: Perholtz discloses capturing a screen image from the 
display of a second computer and transmitting the screen image to a first computer and 
displaying the captured screen image on the display of the first computer. 

Reid et al. US 6,055,964: Reid discloses capturing a digital image, searching for 
predetermined graphical images contained within the digital image, creating a list of the 
predetermined graphical images detected within the digital image. 

Mclntyre et al. US US 2004/0201709 A1 : Mclntyre discloses a client computer scanning 
a digital photograph, recognizing an image in the photograph and creating an identifier 
from the recognized image, and transmitting the identifier to a server. A server 
uploading low resolution copies of images from the client and using the identifier to 
determine other users with whom uploaded images can be shared. 

2. The following is an examiner's statement of reasons for allowance: 

Independent claims 8, 14, 16, 28, 38, 40 and 42 are allowable over the prior art of 
record, specifically: 
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With respect to claim 8, the prior art of record fails to disclose the combination of the 
following method steps: 

searching a bitmap image of a second computer system graphical user 

interface for a first graphical element contained within and comprising less 
than said bitmap image through an automated execution of said first 
computer system commands; 

responsive to said receiving step and results of said searching step, 

generating a user peripheral input device input action within said second 
computer system graphical user interface as interpreted by said second 
computer by automatically creating and passing a signal through a 
communications channel from said first computer system to said second 
computer system graphical user interface; 

monitoring said bitmap image of said second computer system graphical user 
interface automatically from said first computer system for an expected 
second graphical element contained within and comprising less than said 
bitmap image within a predetermined time interval; and 

signaling a failure at said first computer system if said predetermined time 
interval elapses without detecting said expected second graphical 
element. 
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In summary, the allowable subject matter is as follows: 

from a first computer system, searching a bitmap image of a second computer 
system for a graphical element contained within the bitmap image and automatically 
sending a peripheral input device input action through a communications channel to 
the second computer system. 

With respect to claim 14, the prior art of record fails to disclose the combination of the 
following method steps: 

displaying a depiction of said remote system graphical user interface display 
on said local system; 

capturing user input effected in said depiction of said remote system graphical 
user interface display; 

implementing automatically through a local system command language set 
user input emulations that are representative of said captured user input 
when reproduced at said remote computer system graphical user interface 
through a peripheral input device channel; 

image processing said remote computer system graphical displays 

automatically using a local system script that searches for and detects a 
first entity contained within and comprising less than said graphical display 
upon said graphical user interface of said remote computer system; 
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controlling a flow of execution of said local system automatically through a 
scripting language having scripting commands in combination with said 
command language set, said flow which varies responsive to a result of 
detection of said first entity during said image processing step; and 

communicating said user input emulations between said local system and 
said remote computer system graphical user interface through a 
communication interface responsive to said flow controlling step. 

In summary, the allowable subject matter is as follows: 

Displaying a depiction of a remote system graphical user interface on a local system 
and capturing user input effected in the depiction of the remote graphical user 
interface and automatically implementing command language set user input 
emulations representative of the captured user input, reproduced at the remote 
computer system graphical user interface through a peripheral input device channel 
and automatically, image processing the depiction of the remote system graphical 
user interface using a local script that searches for and detects a first entity 
contained within and comprising less than said graphical display upon said graphical 
user interface of said remote computer system and communicating said user input 
emulations between said local system and said remote computer system graphical 
user interface through a communication interface responsive to said flow controlling 
step. 
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Claim 16. A method for using a first computer system to remotely monitor and 
interact with the operation of a second computer system through a graphical user 
interface of said second computer system, comprising the steps of: 

receiving a representation of said second computer system graphical user 
interface at said first computer system; searching said representation of 
said second computer system graphical user interface for a first graphical 
entity contained within and comprising less than said representation of 
said second computer system graphical user interface through an 
automated execution of said first computer system commands; and 
responsive to said receiving step and results of said searching step, 

generating a user peripheral input device input action within said second 
computer system graphical user interface as interpreted by a second 
computer by automatically creating and passing a signal through a 
communications channel from said first computer system to said second 
computer system graphical user interface. 



In summary, the allowable subject matter is as follows: 

from a first computer system, searching a bitmap image of a second computer 
system for a graphical element contained within the bitmap image and automatically 
sending a peripheral input device input action through a communications channel to 
the second computer system. 
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Claim 28. A method for enabling a first computing machine to remotely operate a 
second computing machine through a graphical user interface on said second 
computing machine by using computing scripts that selectively respond to 
changes within graphical displays upon said graphical user interface of said 
second computing machine, comprising the steps of: 

displaying a depiction of said second computing machine graphical user 

interface display on a graphical user interface of said first computing 

machine; 

capturing user input effected in said depiction of said second computing 
machine graphical user interface display; 

image processing said second computing machine graphical displays using a 
first computing machine script that searches for and detects the presence 
of a first entity contained within and comprising less than said graphical 
display upon said graphical user interface of said second computing 
machine; 

controlling a flow of execution of said first computing machine through a 
scripting language having scripting commands in combination with said a 
command language set, said flow which varies responsive to a result of 
detection of said first entity during said image processing step; and 

implementing user input commands at said second computing machine 
graphical user interface emulating said captured user input by 
communicating between said first computing machine and said second 
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computing machine graphical user interface through a communication 
interface responsive to said flow controlling step, a first user input 
command implemented responsive to a detection of said first entity during 
said image processing step and a second user input command different 
from said first user input command implemented responsive to a non- 
detection of said first entity during said image processing step. 



In summary, the allowable subject matter is as follows: 

Displaying a depiction of a computing machine graphical user interface on a local 
computing machine and capturing user input effected in said depiction of said 
second computing machine graphical user interface display and image processing 
said second computing machine graphical displays using a first computing machine 
script that searches for and detects the presence of a first entity contained within 
and comprising less than said graphical display upon said graphical user interface of 
said second computing machine and controlling a flow of execution of said first 
computing machine through a scripting language having scripting commands in 
combination with said a command language set, said flow which varies responsive 
to a result of detection of said first entity during said image processing step and 
implementing user input commands at said second computing machine graphical 
user interface emulating said captured user input by communicating between said 
first computing machine and said second computing machine graphical user 
interface through a communication interface responsive to said flow controlling step. 
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Claim 38. A method for using a first computer system to remotely monitor and 
interact with the operation of a second computer system through a graphical user 
interface of said second computer system, comprising the steps of: 

receiving a bitmap image of said second computer system graphical user 

interface at said first computer system; 
searching said bitmap image of said second computer system graphical user 
interface for a first graphical element contained within and comprising less 
than said bitmap image through an automated execution of said first 
computer system commands; 
responsive to said receiving step and a not-found result of said searching 
step, generating a first user peripheral input device input action within said 
second computer system graphical user interface by automatically 
creating and passing a signal through a communications channel from 
said first computer system to said second computer system graphical user 
interface; 

and responsive to said receiving step and a found result of said searching 
step, generating a second user peripheral input device input action within 
said second computer system graphical user interface different from said 
first user peripheral input device input action by automatically creating and 
passing a signal through an i /o a communications channel from said first 
computer system to said second computer system graphical user 
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interface. 

In summary, the allowable subject matter is as follows: 

receiving a bitmap image of a second computer system graphical user interface at a 
first computer system and searching a bitmap image of a second computer system 
for a graphical element contained within the bitmap image and automatically sending 
a peripheral input device input action through a communications channel to the 
second computer system and responsive to said receiving step and a not-found 
result of said searching step, generating a first user peripheral input device input 
action within said second computer system graphical user interface by automatically 
creating and passing a signal through a communications channel from said first 
computer system to said second computer system graphical user interface and 
responsive to said receiving step and a found result of said searching step, 
generating a second user peripheral input device input action within said second 
computer system graphical user interface different from said first user peripheral 
input device input action by automatically creating and passing a signal through m 
He a communications channel from said first computer system to said second 
computer system graphical user interface. 

Claim 40. A method for enabling a first computing machine to remotely operate a 
second computing machine through a graphical user interface on said second 
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computing machine by using computing scripts that selectively respond to 
changes within graphical displays upon said graphical user interface of said 
second computing machine, comprising the steps of: 

displaying a depiction of said second computing machine graphical user 

interface display on a graphical user interface of said first computing 

machine; 

image processing said second computing machine graphical displays using a 
first computing machine search command set that searches for and 
detects the presence of a first entity contained within and comprising less 
than said graphical display upon said graphical user interface of said 
second computing machine; 

controlling a flow of execution of said first computing machine through a 
scripting language having scripting commands in combination with a 
command language set, said flow which varies responsive to a result of 
detection of said first entity during said image processing step; and 

communicating between said first computing machine and said second 
computing machine graphical user interface through a communication 
interface responsive to said flow controlling step, a first user input 
command implemented at said second computing machine responsive to 
a detection of said first entity during said image processing step and a 
second user input command different from said first user input command 
implemented at said second computing machine responsive to a non- 
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detection of said first entity during said image processing step. 

In summary, the allowable subject matter is as follows: 

displaying a depiction of said second computing machine graphical user interface 
display on a graphical user interface of said first computing machine and image 
processing said second computing machine graphical displays using a first 
computing machine search command set that searches for and detects the presence 
of a first entity contained within and comprising less than said graphical display upon 
said graphical user interface of said second computing machine and capturing user 
input effected in said depiction of said second computing machine graphical user 
interface display and controlling a flow of execution of said first computing machine 
through a scripting language having scripting commands in combination with a 
command language set, said flow which varies responsive to a result of detection of 
said first entity during said image processing step and communicating between said 
first computing machine and said second computing machine graphical user 
interface through a communication interface responsive to said flow controlling step, 
a first user input command implemented at said second computing machine 
responsive to a detection of said first entity during said image processing step and a 
second user input command different from said first user input command 
implemented at said second computing machine responsive to a non-detection of 
said first entity during said image processing step. 
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42. A method for enabling a first computing machine to remotely operate a 
second computing machine through a graphical user interface on said second 
computing machine by using computing scripts that selectively respond to 
changes within graphical displays upon said graphical user interface of said 
second computing machine, comprising the steps of: 

displaying a depiction of said second computing machine graphical user 

interface display on a graphical user interface of said first computing 

machine; 

capturing user input effected in said depiction of said second computing 
machine graphical user interface display; 

developing at least one computing script incorporating user input into said first 
computing machine selected from ones of various scripting functions and 
commands, said at least one computing script further incorporating said 
captured user input; 

image processing said second computing machine graphical displays using a 
first computing machine search command set that searches for and 
detects the presence of a first entity contained within and comprising less 
than said graphical display upon said graphical user interface of said 
second computing machine; 

controlling a flow of execution of said first computing machine through said at 
least one computing script, said flow which varies responsive to a result of 
detection of said first entity during said image processing step; and 
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implementing user input commands at said second computing machine 
graphical user interface emulating said captured user input by 
communicating between said first computing machine and said second 
computing machine graphical user interface through a communication 
interface responsive to said flow controlling step, a first user input 
command implemented at said second computing machine graphical user 
interface responsive to a detection of said first entity during said image 
processing step and a second user input command different from said first 
user input command implemented at said second computing machine 
graphical user interface responsive to a non-detection of said first entity 
during said image processing step. 

In summary, the allowable subject matter is as follows: 

displaying a depiction of said second computing machine graphical user interface 
display on a graphical user interface of said first computing machine; 
capturing user input effected in said depiction of said second computing machine 
graphical user interface display and developing at least one computing script 
incorporating user input into said first computing machine selected from ones of 
various scripting functions and commands, said at least one computing script further 
incorporating said captured user input and image processing said second computing 
machine graphical displays using a first computing machine search command set 
that searches for and detects the presence of a first entity contained within and 
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comprising less than said graphical display upon said graphical user interface of said 
second computing machine and controlling a flow of execution of said first 
computing machine through said at least one computing script, said flow which 
varies responsive to a result of detection of said first entity during said image 
processing step and implementing user input commands at said second computing 
machine graphical user interface emulating said captured user input by 
communicating between said first computing machine and said second computing 
machine graphical user interface through a communication interface responsive to 
said flow controlling step, a first user input command implemented at said second 
computing machine graphical user interface responsive to a detection of said first 
entity during said image processing step and a second user input command different 
from said first user input command implemented at said second computing machine 
graphical user interface responsive to a non-detection of said first entity during said 
image processing step. 

The respective dependent claims add further limitations to the allowable subject matter 
of the independent claims and are, therefore, allowable over the prior art of record. 
Specifically, the prior art of record fails to clearly teach or fairly suggest the combination 
of elements as recited in the claims. 



Conclusion 
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Any comments considered necessary by applicant must be submitted no later 
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